Oz big data meeting. CERN meeting re: datalake.
Resolved hostname problems. Still porting microservices to k8s manifests. Problem with hostnames in api/handler instance.
Dockerised db. Refactored build process and added CI pipeline.
Porting all services to minikube to test cluster locally. Problem at the moment with hostname resolution (grafana can't see db) and also crash loops on the api/handler services which are possibly part of the same problem. Service name needs to match container name (e.g. sdcss-db).
Refactored initialisation, moved all variables to export. Explored creating helm charts from k8s yaml.
Issues: need to merge end-to-end-skeleton with master and rebuild the image. -prod- commands are currently pulling an old version of master where the env variable TRUTH_CATALOGUE_PATH is set instead of SDC1_1400_TRUTH_CATALOGUE_PATH.
Not quite sure how to tell it to always build when there's a new git commit. Do I need to add a tag to the Dockerfile?
Reverted Yan G.'s commit and pushed older image to dockerhub following request.
Started migration from docker-compose to kubernetes. Gitlab now has images for api/handler/grafana.
Refactored SDCSS code & cleaned up deployment.
Looked over SDCSS.
Amended getting started document following Yan's comments.
PR from Yan G. to rucio-client-container.
Looked over the SDCSS.
Looked at 3561: https://github.com/rucio/rucio/issues/3561.
NoScopeFound is not in rucio-admin. Fixed and issued a PR: https://github.com/rucio/rucio/pull/3944
Looked at 2770: https://github.com/rucio/rucio/issues/2770. Looks to have resolved itself. Closed.
Some more useful commands:
Removing the auth token (token is cached):
$ rm -r /tmp/root/.rucio*Started looking at 3561, can see the issue with the exception logic, but can't seem to flush the web service to get dynamic updating after code changes.
Got in touch w/ Mario and Martin Barisits.
They've suggested these issues:
Re: 2770 (From Mario)
for 2770, yeah, that might indeed be an easy one. i think this is
just missing the passing of a parameter somewhere through the stack.
it basically goes like this:
on your client machine to send the request to the server:
bin/rucio -> lib/rucio/client/baseclient.py
then on the server side
lib/rucio/web/rest/authentication.py -> lib/rucio/api/authentication.py
-> lib/rucio/core/authentication.pyNeed to first add an account and associate a userpass identity:
$ rucio-admin account add test_rmb
$ rucio-admin identity add --account test_rmb --type USERPASS --id 'test' --email test@test.com --password testCan list accounts with:
$ rucio-admin account listAnd associated identities with:
$ rucio-admin account list-identities test_rmbOrganised return to work.
Ops & SciOps meetings.
Managed to run rucio unit tests with some bodging inside the container to install pytest:
docker exec -it dev_rucio_1 /bin/bash
git clone https://github.com/rucio/rucio.git
pip install -r /tmp/rucio/etc/pip-requires-test
643 passed, 47 xfailed, 1 xpassed, 231 warnings in 14965.44 seconds Not sure why this is necessary given this should have been installed using the Docker image:
https://hub.docker.com/r/rucio/rucio-dev/dockerfile
Looked through tickets with a view on asking if they can be earmarked (needs to be low priority):
https://github.com/rucio/rucio/issues/1839
Quickly looked over Rohini's external IAM integration for jhub. Fault w/ ESCAPE. Added ticket.
Team day!
Installed Rucio environment again, looked over architecture docs.
Tried to run unit tests and failed. Pytest not installed.
Caught up with emails in morning.
Added feedback (Ghita/Stephane) to Rucio getting started document.
Discussion in ESCAPE-WP2 repository about where to keep Dockerfiles.
Meeting re: features.